{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CIFAR10 Tensorflow Pytorch\n",
    "\n",
    "This notebook walks you through image classification model training in Determined on the popular [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), specifically using the PyTorch machine learning library.  See [this notebook](../cifar10_cnn_tf_keras/CIFAR10-Tensorflow-Keras.ipynb) for the same example built on TensorFlow Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test importing Determined.  In Determined is properly installed, you should see no output.\n",
    "import determined as det"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with the IP address of the Determined master.\n",
    "determined_master = '<master-ip>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run an Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will explore the components of a Determined experiment; namely, the model definition and associated experiment configuration.\n",
    "\n",
    "## Model Directory\n",
    "- `model_def.py`: The PyTorch model definition\n",
    "- `.yaml` configuration files that each govern an individual experiment run\n",
    "\n",
    "Let's look at the contents of the model directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### model_def.py\n",
    "Now drill in and view the model definition file.  Look for the implementation of Determined's `PyTorchTrial` interface.  This is the interface between Determined and PyTorch, which ultimately enables the ML Engineer to leverage Determined's distributed hyperparameter search in a shared runtime without having to worry about these distributed system concerns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat -n model_def.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### const.yaml\n",
    "For our first Determined experiment, we'll run this model training job with fixed hyperparameters.  Note the following sections:\n",
    "- `description`: A short description of the experiment\n",
    "- `data`: A section for user to provide custom key value pairs.  Here we specify where the data resides.  \n",
    "- `hyperparameters`: area for user to define hyperparameters that will be injected into the trial class at runtime. There are constant values for this configuration\n",
    "- `searcher`: hyperparameter search algorithm for the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat -n const.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!det -m {determined_master} experiment create const.yaml ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the experiment completes (which may take a few minutes if Determined agents have to start up), look at the experiment page to see the single completed trial.  Note the validation error around 0.75."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adaptive Hyperparameter Search\n",
    "### adaptive.yaml\n",
    "\n",
    "Next, let's run an experiment with the same model definition, but we'll leverage Determined's adaptive hyperparameter search to efficiently determine the hyperparameter values that yield the lowest validation error.  Note that hyperparameters in the experiment configuration are specified as ranges as opposed to fixed values as in our [first experiment](#const.yaml)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat -n adaptive.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!det -m {determined_master} experiment create adaptive.yaml ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During and after the experiment run, you can view the best (lowest) validation error that Determined's adaptive search finds over time:\n",
    "\n",
    "When the experiment finishes, note that your best performing model achieves a lower validation error than our first experiment that ran with constant hyperparameter values.  From the Determined experiment detail page, you can drill in to a particular trial and view the hyperparameter values used.  You can also access the saved checkpoint of your best-performing model and load it for real-time or batch inference as described in the PyTorch documentation [here](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
